Unfair items detection in educational measurement

Measurement professionals cannot come to an agreement on the definition of the term „item fairness‟. In this paper a continuous measure of item unfairness is proposed. The more the unfairness measure deviates from zero, the less fair the item is. If the measure exceeds the cutoff value, the item is identified as definitely unfair. The new approach can identify unfair items that would not be identified with conventional procedures. The results are in accord with experts‟ judgments on the item qualities. Since no assumptions about scores distributions and/or correlations are assumed, the method is applicable to any educational test. Its performance is illustrated through application to scores of a real test. All researchers without exceptions confess the existence of unfair items, nevertheless Cole & Zieky (2001) indicated: „there is no generally accepted definition of fairness with respect to testing‟. Indeed, the definition should comprise a method for the differentiation between fair and unfair items, a method which is based on some measure. If an appropriate method existed, it would provide a definition of unfair items. Some measurement professionals use the term ‟unfair item‟ in narrow sense implying that such item favors one group of test-takers over another even though the groups are of comparable ability. Differential item functioning method (DIF) verifies whether a test item is biased against particular gender, ethnic, social, or economic groups (e.g. Holland, 1988). DIF tries to assess item functioning by comparison of the item scores for two or more groups. DIF cannot check item fairness on a single group; consequently it cannot present a fairness scale. Speaking about unfair items in broad sense, including ambiguous items and faulty teaching, then item analysis is employed which identifies unfair items with the help of the item discrimination index and item difficulty. Here again an exact measure of item fairness is absent. For example, item analysis cannot tell which of two items is fairer: whose discrimination index equals 0.3 or 0.5? It is also impossible to say which items are ideally fair. Researchers (Zimmerman & Zumbo, 1993; Zimmerman & Williams, 1980; Burton, 2001, 2006; Harrison, 1986) subjected to criticism the conventional methods of test analysis. In what follows, we propose a measure of item unfairness. The method represents each item by means of two parameters, which may be interpreted as coordinates of a point in a plane. In such geometrical presentation fair items are localized near the so-called “ideal line”, whereas unfair items are lying far from it. The farther an item point is from the ideal line, the less fair the corresponding